{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/drive/1br0hM79ORTVNXUpVgkV5t4o4AigGxfwk?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Linear Regression from Scratch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m41136O7L5bV"
      },
      "source": [
        "In this tutorial, we are going to implement a linear regression model to predict california housing prices. We will build the model from scratch using numpy. This will be a great approach to begin understanding regression based models.\n",
        "\n",
        "After completing this tutorial the learner is expected to know the basic building blocks of a linear regression model. The learner is also expected to know the pipeline of reading and transforming data for machine learning workflows.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "Pni17h4R8v8a"
      },
      "outputs": [],
      "source": [
        "## Import the usual libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.datasets import fetch_california_housing\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjyJUfczL4zX"
      },
      "source": [
        "# Importing the dataset\n",
        "\n",
        "The real-world dataset can be obtained by the function `fetch_california_housing` that downloads the dataset for us. \n",
        "\n",
        "The `as_frame` parameter returns a pandas dataframe which is a library useful for viewing contents of the data.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "aOXxbywahC5X",
        "outputId": "24521e0a-6f1a-4e5c-c35d-3abb8112a9af"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-ce5979d0-431a-47c0-a6fb-4bbe7a0e7397\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MedInc</th>\n",
              "      <th>HouseAge</th>\n",
              "      <th>AveRooms</th>\n",
              "      <th>AveBedrms</th>\n",
              "      <th>Population</th>\n",
              "      <th>AveOccup</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "      <th>MedHouseVal</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>8.3252</td>\n",
              "      <td>41.0</td>\n",
              "      <td>6.984127</td>\n",
              "      <td>1.023810</td>\n",
              "      <td>322.0</td>\n",
              "      <td>2.555556</td>\n",
              "      <td>37.88</td>\n",
              "      <td>-122.23</td>\n",
              "      <td>4.526</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8.3014</td>\n",
              "      <td>21.0</td>\n",
              "      <td>6.238137</td>\n",
              "      <td>0.971880</td>\n",
              "      <td>2401.0</td>\n",
              "      <td>2.109842</td>\n",
              "      <td>37.86</td>\n",
              "      <td>-122.22</td>\n",
              "      <td>3.585</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7.2574</td>\n",
              "      <td>52.0</td>\n",
              "      <td>8.288136</td>\n",
              "      <td>1.073446</td>\n",
              "      <td>496.0</td>\n",
              "      <td>2.802260</td>\n",
              "      <td>37.85</td>\n",
              "      <td>-122.24</td>\n",
              "      <td>3.521</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5.6431</td>\n",
              "      <td>52.0</td>\n",
              "      <td>5.817352</td>\n",
              "      <td>1.073059</td>\n",
              "      <td>558.0</td>\n",
              "      <td>2.547945</td>\n",
              "      <td>37.85</td>\n",
              "      <td>-122.25</td>\n",
              "      <td>3.413</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3.8462</td>\n",
              "      <td>52.0</td>\n",
              "      <td>6.281853</td>\n",
              "      <td>1.081081</td>\n",
              "      <td>565.0</td>\n",
              "      <td>2.181467</td>\n",
              "      <td>37.85</td>\n",
              "      <td>-122.25</td>\n",
              "      <td>3.422</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ce5979d0-431a-47c0-a6fb-4bbe7a0e7397')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ce5979d0-431a-47c0-a6fb-4bbe7a0e7397 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ce5979d0-431a-47c0-a6fb-4bbe7a0e7397');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  \\\n",
              "0  8.3252      41.0  6.984127   1.023810       322.0  2.555556     37.88   \n",
              "1  8.3014      21.0  6.238137   0.971880      2401.0  2.109842     37.86   \n",
              "2  7.2574      52.0  8.288136   1.073446       496.0  2.802260     37.85   \n",
              "3  5.6431      52.0  5.817352   1.073059       558.0  2.547945     37.85   \n",
              "4  3.8462      52.0  6.281853   1.081081       565.0  2.181467     37.85   \n",
              "\n",
              "   Longitude  MedHouseVal  \n",
              "0    -122.23        4.526  \n",
              "1    -122.22        3.585  \n",
              "2    -122.24        3.521  \n",
              "3    -122.25        3.413  \n",
              "4    -122.25        3.422  "
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Fetch the data using sklearn function\n",
        "bunch = fetch_california_housing(download_if_missing=True, as_frame=True)\n",
        "\n",
        "# Load the dataframe and view\n",
        "df = bunch.frame\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KUeU_jLylTx7"
      },
      "source": [
        "For this dataset, our target variable is the median house value for California districts, expressed in hundreds of thousands of dollars ($100,000).\n",
        "\n",
        "We can take a closer look at the various statistical parameters of the dataset using pandas. The `describe` function will help us."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "eD4BpBHClgDc",
        "outputId": "e2171ffb-bd38-4e3e-b600-ad3c249a3234"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-bb12ca59-11cc-470b-aa86-ac3f6416cbf8\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MedInc</th>\n",
              "      <th>HouseAge</th>\n",
              "      <th>AveRooms</th>\n",
              "      <th>AveBedrms</th>\n",
              "      <th>Population</th>\n",
              "      <th>AveOccup</th>\n",
              "      <th>Latitude</th>\n",
              "      <th>Longitude</th>\n",
              "      <th>MedHouseVal</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "      <td>20640.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>3.870671</td>\n",
              "      <td>28.639486</td>\n",
              "      <td>5.429000</td>\n",
              "      <td>1.096675</td>\n",
              "      <td>1425.476744</td>\n",
              "      <td>3.070655</td>\n",
              "      <td>35.631861</td>\n",
              "      <td>-119.569704</td>\n",
              "      <td>2.068558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>1.899822</td>\n",
              "      <td>12.585558</td>\n",
              "      <td>2.474173</td>\n",
              "      <td>0.473911</td>\n",
              "      <td>1132.462122</td>\n",
              "      <td>10.386050</td>\n",
              "      <td>2.135952</td>\n",
              "      <td>2.003532</td>\n",
              "      <td>1.153956</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.499900</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.846154</td>\n",
              "      <td>0.333333</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>0.692308</td>\n",
              "      <td>32.540000</td>\n",
              "      <td>-124.350000</td>\n",
              "      <td>0.149990</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>2.563400</td>\n",
              "      <td>18.000000</td>\n",
              "      <td>4.440716</td>\n",
              "      <td>1.006079</td>\n",
              "      <td>787.000000</td>\n",
              "      <td>2.429741</td>\n",
              "      <td>33.930000</td>\n",
              "      <td>-121.800000</td>\n",
              "      <td>1.196000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>3.534800</td>\n",
              "      <td>29.000000</td>\n",
              "      <td>5.229129</td>\n",
              "      <td>1.048780</td>\n",
              "      <td>1166.000000</td>\n",
              "      <td>2.818116</td>\n",
              "      <td>34.260000</td>\n",
              "      <td>-118.490000</td>\n",
              "      <td>1.797000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>4.743250</td>\n",
              "      <td>37.000000</td>\n",
              "      <td>6.052381</td>\n",
              "      <td>1.099526</td>\n",
              "      <td>1725.000000</td>\n",
              "      <td>3.282261</td>\n",
              "      <td>37.710000</td>\n",
              "      <td>-118.010000</td>\n",
              "      <td>2.647250</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>15.000100</td>\n",
              "      <td>52.000000</td>\n",
              "      <td>141.909091</td>\n",
              "      <td>34.066667</td>\n",
              "      <td>35682.000000</td>\n",
              "      <td>1243.333333</td>\n",
              "      <td>41.950000</td>\n",
              "      <td>-114.310000</td>\n",
              "      <td>5.000010</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bb12ca59-11cc-470b-aa86-ac3f6416cbf8')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bb12ca59-11cc-470b-aa86-ac3f6416cbf8 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bb12ca59-11cc-470b-aa86-ac3f6416cbf8');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "             MedInc      HouseAge      AveRooms     AveBedrms    Population  \\\n",
              "count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   \n",
              "mean       3.870671     28.639486      5.429000      1.096675   1425.476744   \n",
              "std        1.899822     12.585558      2.474173      0.473911   1132.462122   \n",
              "min        0.499900      1.000000      0.846154      0.333333      3.000000   \n",
              "25%        2.563400     18.000000      4.440716      1.006079    787.000000   \n",
              "50%        3.534800     29.000000      5.229129      1.048780   1166.000000   \n",
              "75%        4.743250     37.000000      6.052381      1.099526   1725.000000   \n",
              "max       15.000100     52.000000    141.909091     34.066667  35682.000000   \n",
              "\n",
              "           AveOccup      Latitude     Longitude   MedHouseVal  \n",
              "count  20640.000000  20640.000000  20640.000000  20640.000000  \n",
              "mean       3.070655     35.631861   -119.569704      2.068558  \n",
              "std       10.386050      2.135952      2.003532      1.153956  \n",
              "min        0.692308     32.540000   -124.350000      0.149990  \n",
              "25%        2.429741     33.930000   -121.800000      1.196000  \n",
              "50%        2.818116     34.260000   -118.490000      1.797000  \n",
              "75%        3.282261     37.710000   -118.010000      2.647250  \n",
              "max     1243.333333     41.950000   -114.310000      5.000010  "
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6S6WG0Bejxc2"
      },
      "source": [
        "As we can see the data in each of the columns is on different scales. For example, the average bedroom value is around 1 and the average population is around 1425. \n",
        "\n",
        "Generally, machine learing models do not work well when the data is on different scales. Thus, we have to normalize our data in the range [-1,1]. The module [StandardScalar](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) will help us in this.\n",
        "\n",
        "The training data should always be normalized. The testing data should be normalized using the values of the training data. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "pkaOgN44iQLN"
      },
      "outputs": [],
      "source": [
        "# !wget https://raw.githubusercontent.com/Ankit152/Fish-Market/main/Fish.csv\n",
        "# import pandas as pd\n",
        "# df  = pd.read_csv(\"Fish.csv\")\n",
        "# y = df['Weight']\n",
        "# x = df[[\"Length1\", \"Length2\", \"Length3\", \"Height\", \"Width\",\"Weight\"]]\n",
        "\n",
        "df = bunch.frame\n",
        "x = df.iloc[:,:-1] # Select all the columns, except the last column\n",
        "y = df.iloc[:,-1:] # Select the last column\n",
        "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.33, random_state = 1)\n",
        "\n",
        "input_scalar = StandardScaler()\n",
        "output_scalar = StandardScaler()\n",
        "\n",
        "x_train = input_scalar.fit_transform(x_train).T # Normalize train data\n",
        "x_test = input_scalar.transform(x_test).T # Only transform test data using values of train data\n",
        "\n",
        "y_train = output_scalar.fit_transform(y_train).reshape(-1)\n",
        "y_test = output_scalar.transform(y_test).reshape(-1)\n",
        "\n",
        "dataset_copy = [ x_train.copy(), x_test.copy(),  y_train.copy(),  y_test.copy()]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mylVXZDk96a2"
      },
      "source": [
        "# Linear Regression Model\n",
        "\n",
        "Now we define our linear regression model from scratch.\n",
        "\n",
        "A linear regression model is of the form:\n",
        "\n",
        "$y = a_1 x_1 + a_2 x_2 + \\dots + a_nx_n + a_{n+1}$\n",
        "  \n",
        "The above can be rewritten using matrix multiplication as\n",
        "\n",
        "$ y = w^T x $\n",
        "\n",
        "where \n",
        "\n",
        "$ w = [a_1, a_2, \\dots,  a_n, a_{n+1}]^T $\n",
        "\n",
        "$ x = [x_1, x_2, \\dots,  x_n]^T $\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "iJViSowz9nah"
      },
      "outputs": [],
      "source": [
        "class LinearRegression():\n",
        "  def __init__(self, dim, lr = 0.1):\n",
        "    assert isinstance\n",
        "    self.lr = lr\n",
        "    self.w = np.zeros((dim))\n",
        "    self.grads = {\"dw\": np.zeros((dim)) +5}\n",
        "\n",
        "  def forward(self, x):\n",
        "    y = self.w.T @ x\n",
        "    return y\n",
        "  \n",
        "  def backward(self, x, y_hat, y):\n",
        "    assert y_hat.shape == y.shape\n",
        "    self.grads[\"dw\"] = (1 / x.shape[1]) * ((y_hat - y) @ x.T).T\n",
        "    assert self.grads[\"dw\"].shape == self.w.shape\n",
        "    \n",
        "    # print(self.grads[\"dw\"])\n",
        "\n",
        "  def optimize(self):\n",
        "    self.w = self.w - self.lr * self.grads[\"dw\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6UOy32LoqCrL"
      },
      "source": [
        "# Loss\n",
        "\n",
        "For linear regression, various loss functions such as the mean absolute error, mean squared error, or root mean squared error can be used.\n",
        "\n",
        "In this example, we will use the mean squared error (MSE) loss.\n",
        "\n",
        "The MSE loss is given by \n",
        "\n",
        "$ error = \\frac{1}{m} Σ_{i=1}^{m} (y_{true}^{i} - y_{pred}^{i})^2 $ \n",
        "\n",
        "where $i$ denotes the particular obseration/row in the dataset and $m$ is the total number of obserations.\n",
        "\n",
        "To ensure our model is correct, the loss should decrease over each epoch.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3atqq0KirwLu"
      },
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "mgNn3oGjjbxX",
        "outputId": "2f67998e-2fa4-4c43-e091-d5180dd92029"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 0 | Train Loss 0.49999999999999983 | Test Loss 0.43712002508129305\n",
            "Epoch 20 | Train Loss 0.2359010824314295 | Test Loss 0.2325611904818386\n",
            "Epoch 40 | Train Loss 0.22188755162559423 | Test Loss 0.2217635127918686\n",
            "Epoch 60 | Train Loss 0.21474640752415047 | Test Loss 0.2152613227580797\n",
            "Epoch 80 | Train Loss 0.2095989567021037 | Test Loss 0.21037193610067245\n",
            "Epoch 100 | Train Loss 0.20581761895152345 | Test Loss 0.20673038702760732\n",
            "Epoch 120 | Train Loss 0.20303294882659725 | Test Loss 0.20402527473733864\n",
            "Epoch 140 | Train Loss 0.20097918345162274 | Test Loss 0.20201418597478873\n",
            "Epoch 160 | Train Loss 0.19946210112703647 | Test Loss 0.20051638978054404\n",
            "Epoch 180 | Train Loss 0.19833950070910428 | Test Loss 0.19939846923590557\n",
            "Epoch 200 | Train Loss 0.19750719045004836 | Test Loss 0.19856209982480624\n",
            "Epoch 220 | Train Loss 0.1968887724916989 | Test Loss 0.1979347869242644\n",
            "Epoch 240 | Train Loss 0.19642818272324772 | Test Loss 0.19746302071881933\n",
            "Epoch 260 | Train Loss 0.19608424065206406 | Test Loss 0.19710724843033994\n",
            "Epoch 280 | Train Loss 0.19582666642954677 | Test Loss 0.19683818623685323\n",
            "Epoch 300 | Train Loss 0.19563316903289557 | Test Loss 0.19663411372213185\n",
            "Epoch 320 | Train Loss 0.19548731666143612 | Test Loss 0.19647888801910168\n",
            "Epoch 340 | Train Loss 0.19537697849050284 | Test Loss 0.19636048509166582\n",
            "Epoch 360 | Train Loss 0.19529318388491884 | Test Loss 0.19626992725672532\n",
            "Epoch 380 | Train Loss 0.19522928672715056 | Test Loss 0.19620049386222657\n",
            "Epoch 400 | Train Loss 0.19518035283239818 | Test Loss 0.19614713968570663\n",
            "Epoch 420 | Train Loss 0.19514271054504922 | Test Loss 0.1961060658296224\n",
            "Epoch 440 | Train Loss 0.19511362075549118 | Test Loss 0.19607440266698017\n",
            "Epoch 460 | Train Loss 0.19509103436032973 | Test Loss 0.196049975197568\n",
            "Epoch 480 | Train Loss 0.19507341379189 | Test Loss 0.1960311290809374\n",
            "Epoch 500 | Train Loss 0.195059601524607 | Test Loss 0.1960166013981983\n",
            "Epoch 520 | Train Loss 0.19504872305387622 | Test Loss 0.19600542443105118\n",
            "Epoch 540 | Train Loss 0.19504011519474806 | Test Loss 0.19599685384985982\n",
            "Epoch 560 | Train Loss 0.19503327299733605 | Test Loss 0.19599031497728384\n",
            "Epoch 580 | Train Loss 0.19502781036651573 | Test Loss 0.19598536246238987\n",
            "Epoch 600 | Train Loss 0.19502343078313097 | Test Loss 0.19598164992474956\n",
            "Epoch 620 | Train Loss 0.19501990548217418 | Test Loss 0.19597890702760307\n",
            "Epoch 640 | Train Loss 0.19501705714492468 | Test Loss 0.1959769221005947\n",
            "Epoch 660 | Train Loss 0.1950147476759042 | Test Loss 0.1959755289194148\n",
            "Epoch 680 | Train Loss 0.19501286901218934 | Test Loss 0.19597459660841743\n",
            "Epoch 700 | Train Loss 0.1950113361889526 | Test Loss 0.19597402189697258\n",
            "Epoch 720 | Train Loss 0.1950100820879975 | Test Loss 0.19597372315589145\n",
            "Epoch 740 | Train Loss 0.1950090534451732 | Test Loss 0.19597363578501353\n",
            "Epoch 760 | Train Loss 0.1950082078022795 | Test Loss 0.19597370863036628\n",
            "Epoch 780 | Train Loss 0.19500751116991094 | Test Loss 0.19597390118903935\n",
            "Epoch 800 | Train Loss 0.19500693622732843 | Test Loss 0.19597418141927686\n",
            "Epoch 820 | Train Loss 0.19500646092952323 | Test Loss 0.19597452401759977\n",
            "Epoch 840 | Train Loss 0.19500606742426427 | Test Loss 0.1959749090579219\n",
            "Epoch 860 | Train Loss 0.19500574120612196 | Test Loss 0.19597532091250944\n",
            "Epoch 880 | Train Loss 0.19500547045245514 | Test Loss 0.19597574739336301\n",
            "Epoch 900 | Train Loss 0.19500524549975556 | Test Loss 0.1959761790667557\n",
            "Epoch 920 | Train Loss 0.19500505842876395 | Test Loss 0.19597660870438502\n",
            "Epoch 940 | Train Loss 0.1950049027342804 | Test Loss 0.1959770308427642\n",
            "Epoch 960 | Train Loss 0.19500477306123667 | Test Loss 0.1959774414287147\n",
            "Epoch 980 | Train Loss 0.19500466499285687 | Test Loss 0.19597783753361098\n",
            "Epoch 1000 | Train Loss 0.19500457487995757 | Test Loss 0.19597821712271776\n",
            "Epoch 1020 | Train Loss 0.1950044997028894 | Test Loss 0.19597857886881814\n",
            "Epoch 1040 | Train Loss 0.1950044369594934 | Test Loss 0.19597892200155245\n",
            "Epoch 1060 | Train Loss 0.19500438457387914 | Test Loss 0.19597924618562568\n",
            "Epoch 1080 | Train Loss 0.1950043408219371 | Test Loss 0.1959795514224043\n",
            "Epoch 1100 | Train Loss 0.19500430427035154 | Test Loss 0.19597983797049975\n",
            "Epoch 1120 | Train Loss 0.19500427372654458 | Test Loss 0.1959801062817876\n",
            "Epoch 1140 | Train Loss 0.195004248197501 | Test Loss 0.19598035694999025\n",
            "Epoch 1160 | Train Loss 0.19500422685583052 | Test Loss 0.1959805906694935\n",
            "Epoch 1180 | Train Loss 0.1950042090117447 | Test Loss 0.19598080820250163\n",
            "Epoch 1200 | Train Loss 0.195004194089881 | Test Loss 0.1959810103529865\n",
            "Epoch 1220 | Train Loss 0.19500418161010774 | Test Loss 0.1959811979461702\n",
            "Epoch 1240 | Train Loss 0.1950041711716055 | Test Loss 0.19598137181250938\n",
            "Epoch 1260 | Train Loss 0.1950041624396518 | Test Loss 0.1959815327753366\n",
            "Epoch 1280 | Train Loss 0.19500415513463745 | Test Loss 0.19598168164146745\n",
            "Epoch 1300 | Train Loss 0.19500414902293145 | Test Loss 0.19598181919420496\n",
            "Epoch 1320 | Train Loss 0.1950041439092753 | Test Loss 0.19598194618827683\n",
            "Epoch 1340 | Train Loss 0.19500413963044858 | Test Loss 0.19598206334632265\n",
            "Epoch 1360 | Train Loss 0.1950041360499879 | Test Loss 0.19598217135661974\n",
            "Epoch 1380 | Train Loss 0.19500413305378356 | Test Loss 0.19598227087179027\n",
            "Epoch 1400 | Train Loss 0.1950041305464049 | Test Loss 0.19598236250828016\n",
            "Epoch 1420 | Train Loss 0.1950041284480337 | Test Loss 0.19598244684643956\n",
            "Epoch 1440 | Train Loss 0.19500412669190298 | Test Loss 0.19598252443106423\n",
            "Epoch 1460 | Train Loss 0.1950041252221582 | Test Loss 0.1959825957722857\n",
            "Epoch 1480 | Train Loss 0.1950041239920701 | Test Loss 0.19598266134671743\n",
            "Epoch 1500 | Train Loss 0.1950041229625413 | Test Loss 0.19598272159878383\n",
            "Epoch 1520 | Train Loss 0.19500412210085774 | Test Loss 0.19598277694217195\n",
            "Epoch 1540 | Train Loss 0.19500412137964543 | Test Loss 0.19598282776135909\n",
            "Epoch 1560 | Train Loss 0.19500412077599738 | Test Loss 0.19598287441317852\n",
            "Epoch 1580 | Train Loss 0.19500412027074418 | Test Loss 0.19598291722839395\n",
            "Epoch 1600 | Train Loss 0.19500411984784347 | Test Loss 0.1959829565132598\n",
            "Epoch 1620 | Train Loss 0.19500411949386964 | Test Loss 0.19598299255105023\n",
            "Epoch 1640 | Train Loss 0.19500411919758648 | Test Loss 0.1959830256035433\n",
            "Epoch 1660 | Train Loss 0.19500411894959 | Test Loss 0.1959830559124517\n",
            "Epoch 1680 | Train Loss 0.19500411874200962 | Test Loss 0.19598308370079265\n",
            "Epoch 1700 | Train Loss 0.19500411856825786 | Test Loss 0.19598310917419287\n",
            "Epoch 1720 | Train Loss 0.19500411842282128 | Test Loss 0.1959831325221266\n",
            "Epoch 1740 | Train Loss 0.19500411830108505 | Test Loss 0.19598315391908486\n",
            "Epoch 1760 | Train Loss 0.19500411819918667 | Test Loss 0.19598317352567718\n",
            "Epoch 1780 | Train Loss 0.19500411811389318 | Test Loss 0.19598319148966561\n",
            "Epoch 1800 | Train Loss 0.19500411804249856 | Test Loss 0.19598320794693336\n",
            "Epoch 1820 | Train Loss 0.19500411798273784 | Test Loss 0.1959832230223899\n",
            "Epoch 1840 | Train Loss 0.19500411793271513 | Test Loss 0.19598323683081562\n",
            "Epoch 1860 | Train Loss 0.19500411789084354 | Test Loss 0.19598324947764745\n",
            "Epoch 1880 | Train Loss 0.19500411785579488 | Test Loss 0.19598326105971003\n",
            "Epoch 1900 | Train Loss 0.19500411782645727 | Test Loss 0.19598327166589385\n",
            "Epoch 1920 | Train Loss 0.19500411780190013 | Test Loss 0.1959832813777842\n",
            "Epoch 1940 | Train Loss 0.1950041177813444 | Test Loss 0.1959832902702435\n",
            "Epoch 1960 | Train Loss 0.1950041177641382 | Test Loss 0.1959832984119505\n",
            "Epoch 1980 | Train Loss 0.19500411774973558 | Test Loss 0.19598330586589785\n",
            "Epoch 2000 | Train Loss 0.1950041177376798 | Test Loss 0.1959833126898522\n",
            "Epoch 2020 | Train Loss 0.19500411772758847 | Test Loss 0.1959833189367787\n",
            "Epoch 2040 | Train Loss 0.19500411771914136 | Test Loss 0.19598332465523224\n",
            "Epoch 2060 | Train Loss 0.1950041177120707 | Test Loss 0.19598332988971823\n",
            "Epoch 2080 | Train Loss 0.1950041177061521 | Test Loss 0.1959833346810246\n",
            "Epoch 2100 | Train Loss 0.195004117701198 | Test Loss 0.19598333906652787\n",
            "Epoch 2120 | Train Loss 0.19500411769705106 | Test Loss 0.19598334308047438\n",
            "Epoch 2140 | Train Loss 0.1950041176935798 | Test Loss 0.19598334675423898\n",
            "Epoch 2160 | Train Loss 0.19500411769067422 | Test Loss 0.1959833501165632\n",
            "Epoch 2180 | Train Loss 0.19500411768824202 | Test Loss 0.1959833531937735\n",
            "Epoch 2200 | Train Loss 0.19500411768620615 | Test Loss 0.19598335600998243\n",
            "Epoch 2220 | Train Loss 0.19500411768450196 | Test Loss 0.1959833585872731\n",
            "Epoch 2240 | Train Loss 0.19500411768307552 | Test Loss 0.19598336094586813\n",
            "Epoch 2260 | Train Loss 0.19500411768188147 | Test Loss 0.19598336310428585\n",
            "Epoch 2280 | Train Loss 0.19500411768088202 | Test Loss 0.195983365079482\n",
            "Epoch 2300 | Train Loss 0.19500411768004536 | Test Loss 0.19598336688698145\n",
            "Epoch 2320 | Train Loss 0.19500411767934506 | Test Loss 0.1959833685409978\n",
            "Epoch 2340 | Train Loss 0.19500411767875886 | Test Loss 0.19598337005454344\n",
            "Epoch 2360 | Train Loss 0.1950041176782682 | Test Loss 0.1959833714395307\n",
            "Epoch 2380 | Train Loss 0.1950041176778575 | Test Loss 0.1959833727068645\n",
            "Epoch 2400 | Train Loss 0.19500411767751366 | Test Loss 0.19598337386652676\n",
            "Epoch 2420 | Train Loss 0.19500411767722584 | Test Loss 0.1959833749276544\n",
            "Epoch 2440 | Train Loss 0.19500411767698494 | Test Loss 0.1959833758986108\n",
            "Epoch 2460 | Train Loss 0.19500411767678333 | Test Loss 0.19598337678705066\n",
            "Epoch 2480 | Train Loss 0.19500411767661452 | Test Loss 0.19598337759998\n",
            "Epoch 2500 | Train Loss 0.19500411767647324 | Test Loss 0.19598337834381113\n",
            "Epoch 2520 | Train Loss 0.195004117676355 | Test Loss 0.19598337902441254\n",
            "Epoch 2540 | Train Loss 0.195004117676256 | Test Loss 0.1959833796471551\n",
            "Epoch 2560 | Train Loss 0.19500411767617312 | Test Loss 0.19598338021695402\n",
            "Epoch 2580 | Train Loss 0.1950041176761038 | Test Loss 0.19598338073830746\n",
            "Epoch 2600 | Train Loss 0.19500411767604572 | Test Loss 0.19598338121533196\n",
            "Epoch 2620 | Train Loss 0.19500411767599712 | Test Loss 0.19598338165179446\n",
            "Epoch 2640 | Train Loss 0.1950041176759564 | Test Loss 0.19598338205114224\n",
            "Epoch 2660 | Train Loss 0.19500411767592238 | Test Loss 0.19598338241652982\n",
            "Epoch 2680 | Train Loss 0.19500411767589387 | Test Loss 0.19598338275084387\n",
            "Epoch 2700 | Train Loss 0.19500411767587003 | Test Loss 0.1959833830567258\n",
            "Epoch 2720 | Train Loss 0.19500411767585005 | Test Loss 0.19598338333659285\n",
            "Epoch 2740 | Train Loss 0.1950041176758333 | Test Loss 0.19598338359265657\n",
            "Epoch 2760 | Train Loss 0.19500411767581935 | Test Loss 0.19598338382694094\n",
            "Epoch 2780 | Train Loss 0.19500411767580764 | Test Loss 0.19598338404129775\n",
            "Epoch 2800 | Train Loss 0.19500411767579784 | Test Loss 0.19598338423742154\n",
            "Epoch 2820 | Train Loss 0.19500411767578962 | Test Loss 0.19598338441686272\n",
            "Epoch 2840 | Train Loss 0.19500411767578277 | Test Loss 0.19598338458104\n",
            "Epoch 2860 | Train Loss 0.195004117675777 | Test Loss 0.1959833847312515\n",
            "Epoch 2880 | Train Loss 0.1950041176757722 | Test Loss 0.19598338486868488\n",
            "Epoch 2900 | Train Loss 0.19500411767576814 | Test Loss 0.19598338499442705\n",
            "Epoch 2920 | Train Loss 0.19500411767576475 | Test Loss 0.19598338510947227\n",
            "Epoch 2940 | Train Loss 0.19500411767576192 | Test Loss 0.19598338521473044\n",
            "Epoch 2960 | Train Loss 0.1950041176757596 | Test Loss 0.19598338531103396\n",
            "Epoch 2980 | Train Loss 0.19500411767575762 | Test Loss 0.19598338539914462\n",
            "Epoch 3000 | Train Loss 0.19500411767575593 | Test Loss 0.1959833854797592\n",
            "Epoch 3020 | Train Loss 0.19500411767575457 | Test Loss 0.1959833855535154\n",
            "Epoch 3040 | Train Loss 0.1950041176757534 | Test Loss 0.1959833856209966\n",
            "Epoch 3060 | Train Loss 0.19500411767575246 | Test Loss 0.19598338568273665\n",
            "Epoch 3080 | Train Loss 0.19500411767575163 | Test Loss 0.19598338573922397\n",
            "Epoch 3100 | Train Loss 0.19500411767575093 | Test Loss 0.19598338579090527\n",
            "Epoch 3120 | Train Loss 0.19500411767575035 | Test Loss 0.19598338583818956\n",
            "Epoch 3140 | Train Loss 0.19500411767574988 | Test Loss 0.19598338588145087\n",
            "Epoch 3160 | Train Loss 0.1950041176757495 | Test Loss 0.1959833859210314\n",
            "Epoch 3180 | Train Loss 0.19500411767574916 | Test Loss 0.19598338595724435\n",
            "Epoch 3200 | Train Loss 0.1950041176757489 | Test Loss 0.19598338599037615\n",
            "Epoch 3220 | Train Loss 0.19500411767574863 | Test Loss 0.19598338602068902\n",
            "Epoch 3240 | Train Loss 0.19500411767574843 | Test Loss 0.19598338604842275\n",
            "Epoch 3260 | Train Loss 0.19500411767574832 | Test Loss 0.19598338607379676\n",
            "Epoch 3280 | Train Loss 0.19500411767574816 | Test Loss 0.19598338609701185\n",
            "Epoch 3300 | Train Loss 0.19500411767574805 | Test Loss 0.19598338611825167\n",
            "Epoch 3320 | Train Loss 0.19500411767574796 | Test Loss 0.19598338613768423\n",
            "Epoch 3340 | Train Loss 0.19500411767574785 | Test Loss 0.19598338615546346\n",
            "Epoch 3360 | Train Loss 0.1950041176757478 | Test Loss 0.1959833861717299\n",
            "Epoch 3380 | Train Loss 0.1950041176757477 | Test Loss 0.19598338618661224\n",
            "Epoch 3400 | Train Loss 0.19500411767574768 | Test Loss 0.1959833862002283\n",
            "Epoch 3420 | Train Loss 0.19500411767574766 | Test Loss 0.19598338621268585\n",
            "Epoch 3440 | Train Loss 0.19500411767574763 | Test Loss 0.19598338622408337\n",
            "Epoch 3460 | Train Loss 0.1950041176757476 | Test Loss 0.1959833862345112\n",
            "Epoch 3480 | Train Loss 0.1950041176757476 | Test Loss 0.19598338624405162\n",
            "Epoch 3500 | Train Loss 0.19500411767574752 | Test Loss 0.19598338625278036\n",
            "Epoch 3520 | Train Loss 0.19500411767574755 | Test Loss 0.19598338626076636\n",
            "Epoch 3540 | Train Loss 0.19500411767574752 | Test Loss 0.19598338626807282\n",
            "Epoch 3560 | Train Loss 0.19500411767574752 | Test Loss 0.19598338627475767\n",
            "Epoch 3580 | Train Loss 0.1950041176757475 | Test Loss 0.19598338628087364\n",
            "Epoch 3600 | Train Loss 0.1950041176757475 | Test Loss 0.19598338628646922\n",
            "Epoch 3620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338629158868\n",
            "Epoch 3640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338629627257\n",
            "Epoch 3660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338630055787\n",
            "Epoch 3680 | Train Loss 0.1950041176757475 | Test Loss 0.1959833863044786\n",
            "Epoch 3700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338630806564\n",
            "Epoch 3720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338631134748\n",
            "Epoch 3740 | Train Loss 0.1950041176757475 | Test Loss 0.1959833863143501\n",
            "Epoch 3760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338631709722\n",
            "Epoch 3780 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863196106\n",
            "Epoch 3800 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863219101\n",
            "Epoch 3820 | Train Loss 0.19500411767574746 | Test Loss 0.19598338632401394\n",
            "Epoch 3840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338632593876\n",
            "Epoch 3860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338632769982\n",
            "Epoch 3880 | Train Loss 0.19500411767574743 | Test Loss 0.19598338632931098\n",
            "Epoch 3900 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863307851\n",
            "Epoch 3920 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633213377\n",
            "Epoch 3940 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633336768\n",
            "Epoch 3960 | Train Loss 0.19500411767574743 | Test Loss 0.19598338633449658\n",
            "Epoch 3980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633552945\n",
            "Epoch 4000 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633647439\n",
            "Epoch 4020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633733897\n",
            "Epoch 4040 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633812995\n",
            "Epoch 4060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633885368\n",
            "Epoch 4080 | Train Loss 0.19500411767574746 | Test Loss 0.19598338633951579\n",
            "Epoch 4100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634012158\n",
            "Epoch 4120 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863406758\n",
            "Epoch 4140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634118284\n",
            "Epoch 4160 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863416468\n",
            "Epoch 4180 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634207124\n",
            "Epoch 4200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634245957\n",
            "Epoch 4220 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863428149\n",
            "Epoch 4240 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634313991\n",
            "Epoch 4260 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634343732\n",
            "Epoch 4280 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863437094\n",
            "Epoch 4300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634395834\n",
            "Epoch 4320 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863441861\n",
            "Epoch 4340 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863443945\n",
            "Epoch 4360 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634458515\n",
            "Epoch 4380 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634475954\n",
            "Epoch 4400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634491913\n",
            "Epoch 4420 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634506512\n",
            "Epoch 4440 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863451987\n",
            "Epoch 4460 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634532095\n",
            "Epoch 4480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634543272\n",
            "Epoch 4500 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634553503\n",
            "Epoch 4520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634562865\n",
            "Epoch 4540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634571427\n",
            "Epoch 4560 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863457926\n",
            "Epoch 4580 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634586435\n",
            "Epoch 4600 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863459299\n",
            "Epoch 4620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634598989\n",
            "Epoch 4640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634604479\n",
            "Epoch 4660 | Train Loss 0.19500411767574746 | Test Loss 0.195983386346095\n",
            "Epoch 4680 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634614093\n",
            "Epoch 4700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634618298\n",
            "Epoch 4720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634622145\n",
            "Epoch 4740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634625664\n",
            "Epoch 4760 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863462888\n",
            "Epoch 4780 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863463183\n",
            "Epoch 4800 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634634527\n",
            "Epoch 4820 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863463699\n",
            "Epoch 4840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634639245\n",
            "Epoch 4860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634641307\n",
            "Epoch 4880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634643198\n",
            "Epoch 4900 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634644927\n",
            "Epoch 4920 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634646506\n",
            "Epoch 4940 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863464795\n",
            "Epoch 4960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634649273\n",
            "Epoch 4980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634650483\n",
            "Epoch 5000 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634651596\n",
            "Epoch 5020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634652604\n",
            "Epoch 5040 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863465353\n",
            "Epoch 5060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634654378\n",
            "Epoch 5080 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863465516\n",
            "Epoch 5100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634655868\n",
            "Epoch 5120 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634656515\n",
            "Epoch 5140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634657111\n",
            "Epoch 5160 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634657653\n",
            "Epoch 5180 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863465815\n",
            "Epoch 5200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634658605\n",
            "Epoch 5220 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634659024\n",
            "Epoch 5240 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634659407\n",
            "Epoch 5260 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634659754\n",
            "Epoch 5280 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634660073\n",
            "Epoch 5300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634660364\n",
            "Epoch 5320 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466063\n",
            "Epoch 5340 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634660875\n",
            "Epoch 5360 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634661094\n",
            "Epoch 5380 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634661303\n",
            "Epoch 5400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634661489\n",
            "Epoch 5420 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634661658\n",
            "Epoch 5440 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634661816\n",
            "Epoch 5460 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466196\n",
            "Epoch 5480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662088\n",
            "Epoch 5500 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466221\n",
            "Epoch 5520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662318\n",
            "Epoch 5540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662418\n",
            "Epoch 5560 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466251\n",
            "Epoch 5580 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662596\n",
            "Epoch 5600 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662674\n",
            "Epoch 5620 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634662743\n",
            "Epoch 5640 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466281\n",
            "Epoch 5660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634662868\n",
            "Epoch 5680 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466292\n",
            "Epoch 5700 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466297\n",
            "Epoch 5720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663018\n",
            "Epoch 5740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663057\n",
            "Epoch 5760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663093\n",
            "Epoch 5780 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663132\n",
            "Epoch 5800 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466316\n",
            "Epoch 5820 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863466319\n",
            "Epoch 5840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663218\n",
            "Epoch 5860 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863466324\n",
            "Epoch 5880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663262\n",
            "Epoch 5900 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663284\n",
            "Epoch 5920 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663304\n",
            "Epoch 5940 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663318\n",
            "Epoch 5960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663331\n",
            "Epoch 5980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663348\n",
            "Epoch 6000 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663356\n",
            "Epoch 6020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663373\n",
            "Epoch 6040 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663384\n",
            "Epoch 6060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663395\n",
            "Epoch 6080 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663404\n",
            "Epoch 6100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663412\n",
            "Epoch 6120 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663418\n",
            "Epoch 6140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663423\n",
            "Epoch 6160 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663431\n",
            "Epoch 6180 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663437\n",
            "Epoch 6200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663445\n",
            "Epoch 6220 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663448\n",
            "Epoch 6240 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466345\n",
            "Epoch 6260 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466346\n",
            "Epoch 6280 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663462\n",
            "Epoch 6300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663465\n",
            "Epoch 6320 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663465\n",
            "Epoch 6340 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466347\n",
            "Epoch 6360 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863466347\n",
            "Epoch 6380 | Train Loss 0.19500411767574743 | Test Loss 0.1959833863466347\n",
            "Epoch 6400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663479\n",
            "Epoch 6420 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663479\n",
            "Epoch 6440 | Train Loss 0.19500411767574746 | Test Loss 0.1959833863466348\n",
            "Epoch 6460 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663484\n",
            "Epoch 6480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663484\n",
            "Epoch 6500 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663484\n",
            "Epoch 6520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663487\n",
            "Epoch 6540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663487\n",
            "Epoch 6560 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663487\n",
            "Epoch 6580 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663487\n",
            "Epoch 6600 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663492\n",
            "Epoch 6620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663492\n",
            "Epoch 6640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663492\n",
            "Epoch 6660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663495\n",
            "Epoch 6680 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663492\n",
            "Epoch 6700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663495\n",
            "Epoch 6720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663495\n",
            "Epoch 6740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663495\n",
            "Epoch 6760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6780 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6800 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6820 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6900 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6920 | Train Loss 0.19500411767574743 | Test Loss 0.19598338634663498\n",
            "Epoch 6940 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 6980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7000 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7040 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7080 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7120 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7160 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7180 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7220 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7240 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7260 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7280 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7320 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7340 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7360 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7380 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7420 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7440 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7460 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7500 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7560 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7580 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7600 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7680 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7780 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7800 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7820 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7900 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7920 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7940 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 7980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8000 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8040 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8080 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8120 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8160 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8180 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8220 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8240 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8260 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8280 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8320 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8340 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8360 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8380 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8420 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8440 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8460 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8500 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8560 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8580 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8600 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8680 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8780 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8800 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8820 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8900 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8920 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8940 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 8980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9000 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9020 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9040 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9060 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9080 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9100 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9120 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9140 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9160 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9180 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9200 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9220 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9240 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9260 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9280 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9300 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9320 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9340 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9360 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9380 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9400 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9420 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9440 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9460 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9480 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9500 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9520 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9540 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9560 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9580 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9600 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9620 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9640 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9660 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9680 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9700 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9720 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9740 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9760 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9780 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9800 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9820 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9840 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9860 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9880 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9900 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9920 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9940 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9960 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n",
            "Epoch 9980 | Train Loss 0.19500411767574746 | Test Loss 0.19598338634663498\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAbtklEQVR4nO3df5BV5Z3n8fenu6ExgKBI/EGjNBnUQDRN6NWoOxETjeSXWpWkFmIqOMmUmtWY1UkZjRPjkHIrZrbMjrVkDckymdraBBMdJySScTWR0cSoNItBQRkQmdDEHy0qwih0N/3dP+7pzrm3b9O3uy/e5uHzqurinOc8z+nn3NN8+unnnHuuIgIzM0tXXa07YGZmB5eD3swscQ56M7PEOejNzBLnoDczS1xDrTtQ6phjjokZM2bUuhtmZoeUtWvXvhIRU8ttG3VBP2PGDNra2mrdDTOzQ4qkfxtom6duzMwS56A3M0ucg97MLHGjbo7ezA5PXV1dtLe3s3fv3lp3ZVQbN24cTU1NjBkzpuI2DnozGxXa29uZOHEiM2bMQFKtuzMqRQQ7d+6kvb2d5ubmittVNHUjaYGkTZK2SLqhzPbLJHVIejL7+svctsWSNmdfiyvumZkdVvbu3cuUKVMc8gcgiSlTpgz5r55BR/SS6oGlwAVAO7BG0sqI2FhS9a6IuLqk7dHAN4BWIIC1WdvXhtRLMzssOOQHN5zXqJIR/RnAlojYGhGdwArg4gr3fyHwQES8moX7A8CCIfeyAm92dnP7/93Euj/4d4iZWV4lQT8N2J5bb8/KSn1S0npJd0uaPpS2ki6X1CapraOjo8KuF3urcz93/HoLT+3YNaz2ZnZ427lzJy0tLbS0tHDccccxbdq0vvXOzs4Dtm1ra+Oaa64Z9HucffbZ1erukFTrYuzPgR9HxD5JVwD/AHyw0sYRsQxYBtDa2jqiT0Lx56iY2XBMmTKFJ598EoBbbrmFCRMm8JWvfKVve3d3Nw0N5SOztbWV1tbWQb/Ho48+Wp3ODlElI/odwPTcelNW1icidkbEvmz1B8C8SttWi+f2zKzaLrvsMq688krOPPNMrr/+ep544gnOOuss5s6dy9lnn82mTZsAWL16NR//+MeBwi+Jz3/+88yfP5+ZM2dyxx139O1vwoQJffXnz5/Ppz71KU499VQuvfRSej/tb9WqVZx66qnMmzePa665pm+/I1HJiH4NMEtSM4WQXgh8Jl9B0vER8UK2ehHwTLZ8P/BfJR2VrX8YuHHEvTazpP3Nzzew8Y9vVHWfs084km98Ys6Q27W3t/Poo49SX1/PG2+8wSOPPEJDQwMPPvggX/va17jnnnv6tXn22Wd56KGH2L17N6eccgpf/OIX+933vm7dOjZs2MAJJ5zAOeecw29/+1taW1u54oorePjhh2lubmbRokXDPt68QYM+IrolXU0htOuB5RGxQdISoC0iVgLXSLoI6AZeBS7L2r4q6ZsUflkALImIV6vS84H7ezB3b2aHmU9/+tPU19cDsGvXLhYvXszmzZuRRFdXV9k2H/vYx2hsbKSxsZF3vvOdvPTSSzQ1NRXVOeOMM/rKWlpa2LZtGxMmTGDmzJl998gvWrSIZcuWjfgYKpqjj4hVwKqSsptzyzcywEg9IpYDy0fQx4p44sYsHcMZeR8s48eP71v++te/znnnnce9997Ltm3bmD9/ftk2jY2Nfcv19fV0d3cPq061JPesG4/nzexg2bVrF9OmFW4c/OEPf1j1/Z9yyils3bqVbdu2AXDXXXdVZb/JBL2vxZrZwXb99ddz4403Mnfu3IMyAj/iiCP47ne/y4IFC5g3bx4TJ05k0qRJI96vRtucdmtrawzng0def7OTliUP8I1PzOYvzqn8GRBmNjo888wzvPvd7651N2puz549TJgwgYjgqquuYtasWVx77bVFdcq9VpLWRkTZezyTGdH3GmW/t8zMhuT73/8+LS0tzJkzh127dnHFFVeMeJ/JPL1SvhxrZgm49tpr+43gRyq9EX2tO2BmNsqkE/Qe0JuZlZVO0GdG28VlM7NaSybofXulmVl5yVyMNTMbiZ07d/KhD30IgBdffJH6+nqmTp0KwBNPPMHYsWMP2H716tWMHTu2Zo8iPhAHvZkZgz+meDCrV69mwoQJozLo05m6qXUHzCw5a9eu5dxzz2XevHlceOGFvPBC4SG9d9xxB7Nnz+b0009n4cKFbNu2jTvvvJPvfOc7tLS08Mgjj9S458WSG9H7WqxZAn55A7z4VHX3edxp8JFvVVw9IvjSl77Ez372M6ZOncpdd93FTTfdxPLly/nWt77F888/T2NjI6+//jqTJ0/myiuvHPJfAW+XZILeHzxiZtW0b98+nn76aS644AIA9u/fz/HHHw/A6aefzqWXXsoll1zCJZdcUstuViSZoDezhAxh5H2wRARz5szhd7/7Xb9t9913Hw8//DA///nPufXWW3nqqSr/9VFlyczR9wq/N9bMqqCxsZGOjo6+oO/q6mLDhg309PSwfft2zjvvPG677TZ27drFnj17mDhxIrt3765xr8tLJug9cWNm1VRXV8fdd9/NV7/6Vd773vfS0tLCo48+yv79+/nsZz/Laaedxty5c7nmmmuYPHkyn/jEJ7j33nt9Mfbt4IuxZjZSt9xyS9/yww8/3G/7b37zm35lJ598MuvXrz+Y3Rq2dEb0HtKbmZWVTND38oDezKxYMkHv59GbHfr8UMLBDec1SibozezQNm7cOHbu3OmwP4CIYOfOnYwbN25I7Xwx1sxGhaamJtrb2+no6Kh1V0a1cePG0dTUNKQ2FQW9pAXA3wH1wA8iouy7GSR9Ergb+A8R0SZpBvAMsCmr8lhEXDmkHlbIF2PNDm1jxoyhubm51t1I0qBBL6keWApcALQDayStjIiNJfUmAl8GHi/ZxXMR0VKl/g7Kb5gyMytWyRz9GcCWiNgaEZ3ACuDiMvW+CdwG7K1i/8zMbIQqCfppwPbcentW1kfS+4DpEXFfmfbNktZJ+hdJfz78rpqZ2XCM+GKspDrgduCyMptfAE6MiJ2S5gH/JGlORLxRso/LgcsBTjzxxBH1xxdjzcyKVTKi3wFMz603ZWW9JgLvAVZL2ga8H1gpqTUi9kXEToCIWAs8B5xc+g0iYllEtEZEa+9Hdw2VL8aamZVXSdCvAWZJapY0FlgIrOzdGBG7IuKYiJgRETOAx4CLsrtupmYXc5E0E5gFbK36UZiZ2YAGnbqJiG5JVwP3U7i9cnlEbJC0BGiLiJUHaP4BYImkLqAHuDIiXq1Gx0v5nbFmZuVVNEcfEauAVSVlNw9Qd35u+R7gnhH0b8j8rjozs2LJPALBc/RmZuUlE/RmZlZeOkG/dxd3jV1Cc8dDte6JmdmokkzQa38XZ9Y9y/hOPxDJzCwvmaD/E1+MNTPLSyboVZfMoZiZVVV66ejbK83MiiQU9L6/0sysnGSCXr6R3sysrGSCvk/01LoHZmajSjJB7xG9mVl5yQT9n/hirJlZXjJB7xG9mVl5yQR9Hw/ozcyKpBP0HtGbmZWVTtD38ZDezCwvoaDvHdE76M3M8hIK+gI5583MiqQT9NkcvXPezKxYOkGfkaPezKxIQkHvu27MzMpJKOgzfkyxmVmRdIJevuvGzKycdILeUzdmZmVVFPSSFkjaJGmLpBsOUO+TkkJSa67sxqzdJkkXVqPTB+YRvZlZXsNgFSTVA0uBC4B2YI2klRGxsaTeRODLwOO5stnAQmAOcALwoKSTI2J/9Q6h75tVfZdmZimoZER/BrAlIrZGRCewAri4TL1vArcBe3NlFwMrImJfRDwPbMn2d/D4YqyZWZFKgn4asD233p6V9ZH0PmB6RNw31LZZ+8sltUlq6+joqKjj/XlEb2ZWzogvxkqqA24H/mq4+4iIZRHRGhGtU6dOHWGPPKI3M8sbdI4e2AFMz603ZWW9JgLvAVZnH/5xHLBS0kUVtK0ez9GbmZVVyYh+DTBLUrOksRQurq7s3RgRuyLimIiYEREzgMeAiyKiLau3UFKjpGZgFvBE1Y8izwN6M7Mig47oI6Jb0tXA/UA9sDwiNkhaArRFxMoDtN0g6SfARqAbuOqg3HEDeI7ezKy8SqZuiIhVwKqSspsHqDu/ZP1W4NZh9m8YPKQ3M8tL552xfgSCmVlZ6QR9Hwe9mVleQkGfjeid82ZmRRIK+gJfkjUzK5ZO0HuO3sysrHSC3mN5M7OyEgr6Xh7Rm5nlpRP08sVYM7Ny0gn6Pk56M7O8dILeI3ozs7LSCfo+Tnozs7wEg97MzPISDHqP6M3M8pIK+p7wvfRmZqWSCnrAHw5uZlYiqaB3xJuZ9ZdU0IMfhGBmViqpoA9EeFxvZlYkqaBHIAe9mVmRpII+kCfqzcxKJBX0ZmbWX4JB7yG9mVleUkEfvufGzKyfioJe0gJJmyRtkXRDme1XSnpK0pOSfiNpdlY+Q9JbWfmTku6s9gH064tH9GZmRRoGqyCpHlgKXAC0A2skrYyIjblqP4qIO7P6FwG3Awuybc9FREt1u11eIL8x1sysRCUj+jOALRGxNSI6gRXAxfkKEfFGbnU8NZwo94jezKxYJUE/DdieW2/PyopIukrSc8C3gWtym5olrZP0L5L+vNw3kHS5pDZJbR0dHUPofrHAl2LNzEpV7WJsRCyNiHcBXwX+Oit+ATgxIuYC1wE/knRkmbbLIqI1IlqnTp06on54RG9mVqySoN8BTM+tN2VlA1kBXAIQEfsiYme2vBZ4Djh5eF0dnO+6MTPrr5KgXwPMktQsaSywEFiZryBpVm71Y8DmrHxqdjEXSTOBWcDWanR8QL4aa2ZWZNC7biKiW9LVwP1APbA8IjZIWgK0RcRK4GpJ5wNdwGvA4qz5B4AlkrqAHuDKiHj1YBxIgUf0ZmalBg16gIhYBawqKbs5t/zlAdrdA9wzkg4OnUf0ZmZ5fmesmVnikgr6Ao/ozczykgp6R7yZWX9JBT3gtDczK5FU0Bfm6J30ZmZ5SQW9b680M+svsaD3IxDMzEolFfSOeDOz/pIKesCPQDAzK5FU0PsNU2Zm/SUV9OA5ejOzUkkFfTjmzcz6SSrowSN6M7NSyQW9r8WamRVLLuh9OdbMrFhSQe+7bszM+ksq6At6at0BM7NRJamg94jezKy/pIIe8HMQzMxKJBf0vr3SzKxYUkHvqRszs/6SCvoCj+jNzPKSCnqP6M3M+ksq6AG/NdbMrERFQS9pgaRNkrZIuqHM9islPSXpSUm/kTQ7t+3GrN0mSRdWs/OlPKI3M+tv0KCXVA8sBT4CzAYW5YM886OIOC0iWoBvA7dnbWcDC4E5wALgu9n+DiKP6M3M8ioZ0Z8BbImIrRHRCawALs5XiIg3cqvj+VPaXgysiIh9EfE8sCXbn5mZvU0aKqgzDdieW28HziytJOkq4DpgLPDBXNvHStpOK9P2cuBygBNPPLGSfg/I99GbmRWr2sXYiFgaEe8Cvgr89RDbLouI1ohonTp16vD7IMe8mVmpSoJ+BzA9t96UlQ1kBXDJMNuOnO+6MTMrUknQrwFmSWqWNJbCxdWV+QqSZuVWPwZszpZXAgslNUpqBmYBT4y82wPxXTdmZqUGnaOPiG5JVwP3A/XA8ojYIGkJ0BYRK4GrJZ0PdAGvAYuzthsk/QTYCHQDV0XE/oN0LL39PZi7NzM75FRyMZaIWAWsKim7Obf85QO0vRW4dbgdHApHvJlZf4m9M1Y47s3MiiUW9PhirJlZiaSCPpBz3sysRFJBL4GnbszMiiUV9IGc82ZmJZIK+gInvZlZXmJB70cgmJmVSizo8V03ZmYlkgr6QIieWnfDzGxUSSroe1SPwkFvZpaXVNAHctCbmZVIKuh7VOepGzOzEkkFfVDnEb2ZWYmkgr4wovddN2ZmeUkFPdRRd3Afd29mdshJKuh7qMPvjDUzK5ZU0IfqqPMcvZlZkeSC3nfdmJkVSyvokefozcxKJBb0vuvGzKxUUkHv2yvNzPpLKuiRb680MyuVVND79kozs/4qCnpJCyRtkrRF0g1ltl8naaOk9ZJ+Jemk3Lb9kp7MvlZWs/P9O+rbK83MSjUMVkFSPbAUuABoB9ZIWhkRG3PV1gGtEfGmpC8C3wb+U7btrYhoqXK/y+pRHQ2+vdLMrEglI/ozgC0RsTUiOoEVwMX5ChHxUES8ma0+BjRVt5uVCeqod9CbmRWpJOinAdtz6+1Z2UC+APwytz5OUpukxyRdUq6BpMuzOm0dHR0VdKk8v2HKzKy/QaduhkLSZ4FW4Nxc8UkRsUPSTODXkp6KiOfy7SJiGbAMoLW1dfhXUz1Hb2bWTyUj+h3A9Nx6U1ZWRNL5wE3ARRGxr7c8InZk/24FVgNzR9DfA+rxG6bMzPqpJOjXALMkNUsaCywEiu6ekTQX+B6FkH85V36UpMZs+RjgHCB/EbeqpHrk++jNzIoMOnUTEd2SrgbuB+qB5RGxQdISoC0iVgJ/C0wAfioJ4A8RcRHwbuB7knoo/FL5VsndOlWlunqP6M3MSlQ0Rx8Rq4BVJWU355bPH6Ddo8BpI+ngUKjOHyVoZlYqqXfGqq6OOjx1Y2aWl1TQR30jY+lmf4+nb8zMeqUV9A1HMI59dHZ7+sbMrFdSQd8z5giOoNNBb2aWk1TQM+YdjNF+9nXurXVPzMxGjeSCHqDzrT017oiZ2eiRVNBrrIPezKxUUkHfeMR4APbs3lXjnpiZjR5pBf2RxwKw9/WXatwTM7PRI6mgP+LoEwDo3vVijXtiZjZ6JBX0E44pPCa/Z7eD3sysV1JBP/6oY+mKerTHQW9m1iupoFddPa/UTWHsnn6PyzczO2wlFfQAr4ydxqS3tg9e0czsMJFc0O8ZfxLHdv+x1t0wMxs1kgv6rsnNTGIPe3cN/0PGzcxSklzQj5k6C4CXt22ocU/MzEaH5IL+mBnvAeC1bb+vcU/MzEaH5IL+xD+bwxvxDuKP62rdFTOzUSG5oG8c08CWhllMfs1TN2ZmkGDQA+ycNIdpnVuJzjdr3RUzs5pLMujrZn6AMXTz8voHa90VM7OaSzLom+ddwJvRyK6n7qt1V8zMai7NoD9uCmvqWzh2+z9Dd2etu2NmVlMVBb2kBZI2Sdoi6YYy26+TtFHSekm/knRSbttiSZuzr8XV7PwB+suLsxYxqed1dq+7++34lmZmo9agQS+pHlgKfASYDSySNLuk2jqgNSJOB+4Gvp21PRr4BnAmcAbwDUlHVa/7A5v3wU/ybM909j+4BHxR1swOY5WM6M8AtkTE1ojoBFYAF+crRMRDEdGbpo8BTdnyhcADEfFqRLwGPAAsqE7XD+zPjj2S+0+6jiP3vsjuFV/wFI6ZHbYqCfppQP5xkO1Z2UC+APxyKG0lXS6pTVJbR0f1nlGz8NOf4Tt1n2Pi1lXsvfOD8Owq6Npbtf2bmR0KGqq5M0mfBVqBc4fSLiKWAcsAWltbo1r9OfbIcVzwhSV8ZflUruv4e05YsYj9dY10H/Uu6o6eQcP4yahxEowZB6qHuvrs37rCv2Zmb6eJx0PLoqrvtpKg3wFMz603ZWVFJJ0P3AScGxH7cm3nl7RdPZyODtfpTZO54brrWfrAx3np9//M3M71vOvlPzKtYz1H6k0m6i0a6aKOHurpeTu7ZmZW5Plx76b5IAS9Ig48gJbUAPwr8CEKwb0G+ExEbMjVmUvhIuyCiNicKz8aWAu8Lyv6f8C8iHh1oO/X2toabW1twzuaQXR29/CvL+1m88u7eWV3Jzv/vZN93fvp2t9DV3fQtX8/PT37qaMH9fTQ+8rkX6H8yxX9FiCo2h8kZnaYOWnKeP7qo6cPq62ktRHRWm7boCP6iOiWdDVwP1APLI+IDZKWAG0RsRL4W2AC8FNJAH+IiIsi4lVJ36TwywFgyYFC/mAb21DHe6ZN4j3TJtWqC2Zmb7tBR/Rvt4M5ojczS9WBRvRJvjPWzMz+xEFvZpY4B72ZWeIc9GZmiXPQm5klzkFvZpY4B72ZWeJG3X30kjqAfxvBLo4BXqlSdw4Vh9sxH27HCz7mw8VIjvmkiJhabsOoC/qRktQ20JsGUnW4HfPhdrzgYz5cHKxj9tSNmVniHPRmZolLMeiX1boDNXC4HfPhdrzgYz5cHJRjTm6O3szMiqU4ojczsxwHvZlZ4pIJekkLJG2StEXSDbXuz0hImi7pIUkbJW2Q9OWs/GhJD0janP17VFYuSXdkx75e0vty+1qc1d8saXGtjqkSkuolrZP0i2y9WdLj2XHdJWlsVt6YrW/Jts/I7ePGrHyTpAtrcySVkTRZ0t2SnpX0jKSzDoNzfG32M/20pB9LGpfaeZa0XNLLkp7OlVXtvEqaJ+mprM0dyj7t6YAi4pD/ovDJV88BM4GxwO+B2bXu1wiO53jgfdnyRAof5Tgb+DZwQ1Z+A3BbtvxR4JeAgPcDj2flRwNbs3+PypaPqvXxHeC4rwN+BPwiW/8JsDBbvhP4Yrb8n4E7s+WFwF3Z8uzs3DcCzdnPRH2tj+sAx/sPwF9my2OBySmfY2Aa8DxwRO78XpbaeQY+QOHjU5/OlVXtvAJPZHWVtf3IoH2q9YtSpRf2LOD+3PqNwI217lcVj+9nwAXAJuD4rOx4YFO2/D1gUa7+pmz7IuB7ufKieqPpi8IHx/8K+CDwi+yH+BWgofQcU/hYy7Oy5YasnkrPe77eaPsCJmWhp5LylM/xNGB7Fl4N2Xm+MMXzDMwoCfqqnNds27O58qJ6A32lMnXT+wPUqz0rO+Rlf67OBR4Hjo2IF7JNLwLHZssDHf+h9Lr8d+B6oCdbnwK8HhHd2Xq+733HlW3fldU/lI63GegA/j6brvqBpPEkfI4jYgfw34A/AC9QOG9rSfs896rWeZ2WLZeWH1AqQZ8kSROAe4D/EhFv5LdF4dd5EvfGSvo48HJErK11X95GDRT+vP+fETEX+HcKf9L3SekcA2Tz0hdT+CV3AjAeWFDTTtVALc5rKkG/A5ieW2/Kyg5ZksZQCPn/ExH/mBW/JOn4bPvxwMtZ+UDHf6i8LucAF0naBqygMH3zd8BkSQ1ZnXzf+44r2z4J2Mmhc7xQGIm1R8Tj2frdFII/1XMMcD7wfER0REQX8I8Uzn3K57lXtc7rjmy5tPyAUgn6NcCs7Or9WAoXblbWuE/Dll1F/1/AMxFxe27TSqD36vtiCnP3veWfy67gvx/Ylf2ZeD/wYUlHZaOpD2dlo0pE3BgRTRExg8K5+3VEXAo8BHwqq1Z6vL2vw6ey+pGVL8zu1mgGZlG4cDXqRMSLwHZJp2RFHwI2kug5zvwBeL+kd2Q/473HnOx5zqnKec22vSHp/dlr+LncvgZW64sWVbz48VEKd6c8B9xU6/6M8Fj+I4U/7dYDT2ZfH6UwP/krYDPwIHB0Vl/A0uzYnwJac/v6PLAl+/qLWh9bBcc+nz/ddTOTwn/gLcBPgcasfFy2viXbPjPX/qbsddhEBXcj1PhYW4C27Dz/E4W7K5I+x8DfAM8CTwP/m8KdM0mdZ+DHFK5BdFH4y+0L1TyvQGv2+j0H/A9KLuiX+/IjEMzMEpfK1I2ZmQ3AQW9mljgHvZlZ4hz0ZmaJc9CbmSXOQW9mljgHvZlZ4v4/Fb6sr7kqoKMAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "num_epochs = 10000\n",
        "train_loss_history = []\n",
        "test_loss_history = []\n",
        "w_history = []\n",
        "dim = x_train.shape[0]\n",
        "num_train = x_train.shape[1]\n",
        "num_test = x_test.shape[1]\n",
        "\n",
        "\n",
        "model = LinearRegression(dim = dim, lr = 0.1)\n",
        "for i in range(num_epochs):\n",
        "  y_hat = model.forward(x_train)\n",
        "  train_loss = 1/(2 * num_train) * ((y_train - y_hat) ** 2).sum()\n",
        "\n",
        "  w_history.append(model.w)\n",
        "  model.backward(x_train,y_hat,y_train)\n",
        "  model.optimize()\n",
        "\n",
        "  y_hat = model.forward(x_test)\n",
        "  test_loss = 1/(2 * num_test) * ((y_test - y_hat) ** 2).sum()\n",
        "\n",
        "  train_loss_history.append(train_loss)\n",
        "  test_loss_history.append(test_loss)\n",
        "\n",
        "  if i % 20 == 0:\n",
        "    print(f\"Epoch {i} | Train Loss {train_loss} | Test Loss {test_loss}\")\n",
        "\n",
        "plt.plot(range(num_epochs), train_loss_history, label = \"Training\")\n",
        "plt.plot(range(num_epochs), test_loss_history, label = \"Test\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ldFMBPuvr0l0"
      },
      "source": [
        "# Results\n",
        "\n",
        "Before viewing the results, we need to reverse the transformations applied on the output variable y.\n",
        "\n",
        "The `inverse_transform` method of the StandardScaler object will help us."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZycI4aExMsoC",
        "outputId": "47c6b8fa-d1ee-4ff6-90d0-7e9289cfe40e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Test Set Error 0.5263803029005855\n"
          ]
        }
      ],
      "source": [
        "from sklearn.metrics import mean_squared_error\n",
        "y_test = output_scalar.inverse_transform(y_test[np.newaxis,:])\n",
        "y_hat  = output_scalar.inverse_transform(y_hat[np.newaxis,:])\n",
        "error = (((y_test - y_hat) ** 2).sum() / num_test )\n",
        "print(\"Test Set Error\", error)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UoYobRS9uBIv"
      },
      "source": [
        "# Libraries\n",
        "\n",
        "Instead of coding everything from scratch, i.e the model, loss functions, and gradient calculations, there are many libaries that have implemented many machine learning algorithms for us.\n",
        "\n",
        "These libraries will generally be faster and more optimized. We can use the LinearRegression and SGD regressor module from scikit learn to compare our model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "txWBY_0eoNN_",
        "outputId": "7886b7e0-c383-4676-e825-3d7197d06d80"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Test Set Error 0.5892243304217802\n"
          ]
        }
      ],
      "source": [
        "from sklearn.linear_model import SGDRegressor\n",
        "\n",
        "\n",
        "x_train, x_test, y_train, y_test = dataset_copy\n",
        "sgd = SGDRegressor()\n",
        "sgd.fit(x_train.T, y_train)\n",
        "y_hat = sgd.predict(x_test.T)\n",
        "y_test = output_scalar.inverse_transform(y_test[np.newaxis,:])\n",
        "y_hat  = output_scalar.inverse_transform(y_hat[np.newaxis,:])\n",
        "error = mean_squared_error(y_test, y_hat, squared = True)\n",
        "print(\"Test Set Error\", error)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9CaqphG8TG7V",
        "outputId": "579bc8dd-6093-4155-8966-f8c60c34b1b3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Test Set Error 0.5263803029005857\n"
          ]
        }
      ],
      "source": [
        "from sklearn.linear_model import LinearRegression as LR\n",
        "\n",
        "x_train, x_test, y_train, y_test = dataset_copy\n",
        "lr = LR()\n",
        "lr.fit(x_train.T, y_train)\n",
        "y_hat = lr.predict(x_test.T)\n",
        "y_test = output_scalar.inverse_transform(y_test[np.newaxis,:])\n",
        "y_hat  = output_scalar.inverse_transform(y_hat[np.newaxis,:])\n",
        "error = mean_squared_error(y_test, y_hat, squared = True)\n",
        "print(\"Test Set Error\", error)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyODV4REEuhLrJ1l8OWl6dFt",
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Linear Regression.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
